Technology Under Consideration for ISO/IEC 23090-32

ISO/IEC 23090-31 [1] defines a scalable haptics coding framework supporting up to 65536 channels per perception through a 16-bit channel identifier (channel_id).

ISO/IEC 23090-32 [2] specifies the carriage of haptics data in ISOBMFF. However, in its current form, it constrains the channel identifier to 8 bits in several structures. This contribution proposes an alignment of the channel identifier bit-width. == Problem Statement In ISO/IEC 23090-31, the haptic channel identifier is defined as a 16-bit unsigned integer (16 uimsbf) in both metadata and data syntax.

In ISO/IEC 23090-32, the channel identifier is limited to 8 bits in multiple structures, including: • HapticChannelDescriptionHeaderBox • HapticBandConfigurationBox • MIHSBandGroupBox

This discrepancy leads to the following issues:

As a result, interoperability between encoders, decoders, reference software, and delivery systems cannot be guaranteed.

ISO/IEC 23090-32 [2] specifies the carriage of this haptics data in the ISO Base Media File Format (ISOBMFF), including both single-track and multi-track encapsulation modes. In the multi-track mode, a MIHS track may reference one or more MIHS band tracks that carry the haptic data samples associated with the experience.

Clause 5.2 of ISO/IEC 23090-32 [2] currently specifies the following requirements for referenced MIHS band tracks:

The sample entries of each referenced MIHS band track shall satisfy the following conditions: a) It shall contain all the necessary metadata in its configuration box needed for decoding the samples of this band track without relying on the MIHS track. b) Its configuration box and haptic experience boxes shall not contradict the information in the same boxes of the MIHS track that is referencing this MIHS band track. c) Its configuration box and haptic experience boxes shall not contain any information about the samples in other referenced MIHS band tracks […​] However, these requirements introduce ambiguity and potential inconsistency with the overall architecture defined in the specification.

In particular: • The specification defines that, in multi-track mode, the MIHS track may reference MIHS band tracks that carry the samples of the experience, while the MIHS track provides global configuration and metadata. • The MIHSConfigurationBox and associated description boxes in the MIHS track reflect initialization units and provide decoder configuration for the haptic experience.

In this context, requirement (a) implies that each MIHS band track shall be independently decodable without relying on the MIHS track. This contradicts the intended separation of responsibilities between: the MIHS track, which provides global experience-level metadata, and the MIHS band tracks, which carry band-specific data and associated metadata.

As a result: • Requirement (a) may incorrectly suggest full independence of MIHS band tracks • Requirements (b) and (c) do not clearly define the expected scope of metadata in band tracks versus the MIHS track • The current wording may lead to inconsistent implementations regarding metadata duplication and dependency handling

The specification ISO/IEC 23090-32 [2] defines that: • A MIHS track may either contain all samples (single-track mode) or reference MIHS band tracks that contain the samples (multi-track mode) • Metadata describing the haptic experience, perceptions, channels, and bands originates from the MIHS stream and is reflected in configuration and description boxes

In multi-track mode: • The MIHS track acts as the entry point and global description of the experience • MIHS band tracks provide band-level data and associated metadata • Track referencing and grouping mechanisms are used to associate band tracks with channels and perceptions

Therefore, requiring that each MIHS band track be decodable “without relying on the MIHS track” is not aligned with: • the referencing model • the distribution of metadata across tracks • the intended decoding workflow combining MIHS track and band tracks

It is proposed to revise the requirements on referenced MIHS band tracks to clarify the dependency model and avoid ambiguity.The sub-clause for which the changes are proposed is 5.2 MIHS streams and tracks ISO/IEC 23090-32 [2]

The following conditions apply to ensure consistency between the MIHS track and the referenced MIHS band tracks.

The current wording of the requirements for referenced MIHS band tracks introduces ambiguity regarding metadata ownership and decoding dependencies. The proposed changes clarify that MIHS band tracks contain band-specific metadata while relying on the MIHS track for global information, ensuring consistency with the overall design of ISO/IEC 23090-32

Haptics as a media type is an emerging modality that enhances the realism of the overall user experience. It can play a subtle but important role in shaping the user’s experience. It acts as a third sensory channel alongside sight and sound. Haptics introduces tactile signals making the entire user experience engaging and immersive. A haptic movie trailer [1] was released on Apple TV app on iPhone that played synchronized vibrations that matched the scene in the movie trailer.

Another key role is in accessibility. For users with visual or auditory impairments, haptic media data can act as primary signals guiding navigation, confirming actions, or alerting users to changes. For such users, haptic media becomes an alternative way of experiencing the content.

The key to achieving a rich experience with haptic media along with audio-visual is the synchronization with audio-visual media. With improper synchronization, the experience can feel poor and uncomfortable to enjoy the content as it is with audio-visual synchronization.

One common way Haptic format is described is using human-readable textual form. In MPEG, HJIF format [2] has been described which is developing using JSON providing haptic information. HJIF describes properties which when provided to a HJIF decoder synthesizes a haptic signal accordingly.

Currently, ISO/IEC 23090-32 [4] defines the encapsulation of binary MIHS unit data in ISOBMFF. There is a gap in the specification on how haptic content described in textual format can be encapsulation in ISOBMFF container.

The Haptics JSON Interchange Format (HJIF), defined in ISO/IEC 23090-31 [2], provides a structured and human-readable way to represent haptic media. It organizes haptic content using a hierarchical model. At the top level, an HJIF document defines an experience, which represents a complete haptic scene and captures the overall structure of the content. Within an experience, haptic data is grouped into one or more perceptions, each corresponding to a distinct type or modality of haptic sensation.

Each perception is further divided into channels, which represent logical streams of haptic information. A channel may map to a specific target, such as a particular body location or a group of actuators. Channels are then subdivided into bands, which capture different perceptual components of the haptic signal.

Within each band, the content is described using effects, which are the fundamental time-based units of haptic media. Effects are defined by temporal properties such as start time and duration, along with perceptual parameters like intensity, frequency, magnitude, or other modality-specific attributes. They may also include parametric descriptions, such as envelopes or modulation curves, enabling the haptic signal to evolve dynamically over time.

The timing information for a haptic signal is described with band and effects in HJIF. The ‘position’ property indicates the time offset of the effect in a band. Within each effect, keyframes are included. A keyframe has a property ‘relative_position’ which indicates the relative time of the keyframe with respect to the parent effect.

HJIF format is defined as interchange file format and MIHS format is defined as transport format in ISO/IEC 23090-31[1]. MIHS carriage format in ISOBMFF is defined in ISO/IEC 23090-32[2].

We still have concerns about the potential market fragmentation and bitrate efficiency by specifying new text-based haptics carriage format in ISOBMFF, which was described in File Format BoG minutes at MPEG#154 meeting. We think the use cases and the requirements should be clarified for the standardization of Text-based haptics HJIF format encapsulation in ISOBMFF in WG3 and WG7.

However, even if the use cases and the requirements for Text-based haptics HJIF format encapsulation in ISOBMF are not clear, we would investigate the feasibility of text-based haptics HJIF format encapsulation in ISOBMFF to have efficient discussion on these topics.

Response: A class of haptic content is expressed in textual, structured form. Rather than carrying binary-encoded actuator signals, these formats use human-readable data models primarily JSON to describe time-sequenced haptic events, parametric properties, and modulation curves. Prominent examples include:

Some properties such as content encoding, content format, and timescale are fundamental properties that are common across haptic data formats because they provide the essential foundation for representing, exchanging, and interpreting haptic content. Content encoding specifies how the file is represented as a sequence of bytes (e.g., UTF-8), ensuring that the content can be reliably stored, transmitted, and parsed across different platforms and software implementations. Content format, such as JSON, defines the structural organization of the data, enabling tools and playback engines to consistently interpret metadata, timing information, and haptic primitives regardless of the underlying haptic specification. Timescale establishes the temporal reference for all haptic events, allowing playback systems to reproduce effects with accurate timing and synchronization. These properties are independent of any particular haptic codec or effect model and constitute the core infrastructure required by any haptic interchange format, making them natural candidates for standardization across diverse haptic ecosystems.

A generic text-based haptic sample entry will factorize common aspects (text encoding, content format) while leaving interpretation to the identified payload format. This approach improves extensibility and allows future text-based haptic formats to be supported without introducing new ISOBMFF sample entry definitions. Therefore, we believe it is limiting to only describe the carriage of only one of the formats in ISOBMFF

This contribution proposes a generic ISOBMFF carriage layer for descriptive text-based haptics with common properties. The proposed layer defines how text-based haptic documents are transported and how the decoder identifies the corresponding document format. The interpretation of the document remains entirely format-specific. Each descriptive haptics format remains independently specified and preserves its own syntax and semantics.

Response: The ISO Base Media File Format (ISOBMFF) [6] is not limited to serving as a distribution format for delivering finalized media content to end users. It also provides a flexible foundation for production and authoring workflows. ISO/IEC 23090-31[1] defines two formats to describe haptics data MIHS and HJIF. ISOBMFF binding for MIHS is already described in ISO/IEC 23090-32[5]. Thus, defining ISOBMFF binding for HJIF either for production workflow or even distribution is within the working principle of ISOBMFF.

Section HJIF haptic sample entry describes the sample entry of an HJIF haptic track. Section HJIF sample format describes the sample format for the HJIF haptic track.

The proposed carriage model stores each Spatial Unit as an individual ISOBMFF Item. The payload of each Item contains one complete Spatial Unit as defined by ISO/IEC 23090-32, while Temporal Units continue to be transported unchanged in timed MIHS samples.

This approach clearly separates timed media from non-timed spatial configuration information. Spatial Units become standalone ISOBMFF Items that are resolved during decoder initialization before the beginning of Temporal Unit.

The Figure 1 illustrates this carriage architecture.

Figure 1: haptic carriage architecture

The ISOBMFF entity grouping mechanism provides the way to associate multiple entities (tracks and/or items) with the same haptic experience.

A new grouping_type ‘hspt’ identifies a group composed of one or more MIHS tracks and one or more Spatial Items.

During file initialization, the decoder identifies the EntityToGroupBox entries with the grouping type ‘hspt’. For each ‘hspt’ group, the decoder determines the associated MIHS track(s) and the referenced Spatial Item(s). Each Spatial Item is then retrieved, parsed, and stored in a local Spatial Unit table indexed by its Item identifier and associated with the corresponding MIHS track(s).

During playback, the decoder resolves these identifiers using the Spatial Unit table constructed during initialization before rendering the corresponding haptic content.

It is proposed to agree on the carriage mechanism presented in clause 2 for an amendment of the ISO/IEC 23090-32 specifications.

[1] WG03N01412, “Text of ISO/IEC FDIS 23090-32 Carriage of Haptics Data”, MPEG#148, Kemer, Türkiye, November 2024.